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DETAILED ACTION 
Response to Amendment 
The amendment filed on 11/20/2003 has been entered. Claims 1, 10, 19 and 31have been 
amended and Claim 32 has been newly added. Claims 1, 3, 4, 6-10, 12-19, 24-29, 31, and 32 are 
pending in the present application. 

Claim Rejections - 35 USC § 103 

1. The following is a quotation of 35 U.S. C 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claim 1,3,4,6-10,12-19,24-29 and 31 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ohshima et al. U.S. Patent No. 6,522,312 (hereinafter Ohshima) in view of 
Latypov U.S. Patent No. 6,624,853 (Latypov) and Sato U.S. Patent No. 6,445,815 (hereinafter 
Sato). 

3. Claim 1: 

(1) Ohshima teaches an augmented reality presentation apparatus for superimposing a 
virtual object in a real space, characterized by comprising: 

Objective viewpoint augmented reality presentation means for superimposing the virtual 
object viewed from a first viewpoint position, which differs from any player's viewpoint 
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position, in the real space viewed from the first viewpoint position (e.g., figure 2, 7; column 7-8; 
column 14-15), wherein said objective viewpoint augmented reality presentation means includes 

First video sensing means for sensing a video of the real space viewed from the first 
viewpoint position (e.g., camera or CCD 230; figure 2 and 7; column 7-8; column 14-15); 

First video generation means for generating a video of the virtual object viewed from the 
first viewpoint position (e.g., figure 2 and 7; column 7-8; column 14-17); 

First video composition means for composing an augmented reality video viewed from 
the first viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the first viewpoint position (e.g., figure 2 and 7; column 7-8; column 14-15), and 

Objective viewpoint video display means for displaying the augmented reality video 
obtained from said first video composition means (e.g., figure 2 and 7; column 7-8; column 14- 
15); 

wherein said apparatus further comprises: 

Player's viewpoint augmented reality presentation means for superimposing the virtual 
object viewed from the player's viewpoint position in the real space viewed from the player's 
viewpoint position (e.g., figure 2 and 7; column 3-6; column 11-13); 

wherein said player's viewpoint augmented reality presentation means includes 

Second video sensing means for sensing a video of the real space viewed from the 
player's viewpoint position (e.g., figure 2 and 7; column 3-6; column 11-13); 

Player's viewpoint position acquiring means for acquiring information indicating the 
player's viewpoint position (e.g., figure 2 and 7; column 3-6; column 11-13); 
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Second video generation means for generating a video of the virtual object viewed from 
the player's viewpoint position using the information indicating the player's view position (e.g., 
figure 2 and 7; column 3-6; column 1 1-13); 

Second video composition means for composing an augmented reality video viewed from 
the player's viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the player's viewpoint position (e.g., figure 2 and 7; column 3-6; column 1 1-13); 
and 

Display means for displaying to the player the augmented reality video viewed from the 
player's viewpoint position (e.g., figure 2 and 7; column 3-6; column 11-13). 

(2) However, it is not clear whether Ohshima teaches implicitly display means for 
displaying the augmented reality videos on a predetermined display apparatus that is separate 
from THE player's display apparatus . 

(3) Latypov teaches display means for displaying the augmented reality videos including 
displaying the image in virtual space on a predetermined display apparatus and the superimposed 
image from the real space and virtual space on a separate display apparatus (e.g., Latypov 
column 11). 

(4) Ohshima at least suggests display means for displaying the augmented reality videos 
on a predetermined display apparatus (displaying the image on a table in a real environment) that 
is separate from any player's display apparatus (See Ohshima figure 7; column 1 1, lines 50-65; 
column 12-13) and Ohshima teaches display means for displaying the augmented reality videos 
on a display apparatus (displaying the image on a table in a real environment or displaying the 
mixed reality image on other player's display apparatus or displaying the mixed reality image on 
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the left/right-hand side HMD) that is separate from ONE of the player's display apparatus (See 
Ohshima figure 7; column 11, lines 50-65; column 12-13). 

(5) One having the ordinary skill in the art would have been motivated to do this because 
it would have provided a separate display device for viewing by the players. 

Claim 3: 

The apparatus according to claim J, characterized in that said player 's viewpoint 
augmented reality presentation means further comprises: the second video generation means for 
generating a video of the virtual object viewed from said player 's viewpoint position; and the 
display means for displaying to the player the video of the virtual object viewed from said 
player 's viewpoint position on a display surface through which the player can visually see the 
real space. 

Claim 3 recites all the limitations of claim 1 and adds the limitation of "the second video 
generation means" and "the display means " Ohshima teaches a video generation means and 
display means (e.g., figure 2 and 7; column 3-6; column 11-13). 

Claim 4: 

The apparatus according to claim ], characterized by further comprising information 
generation means for generating information that pertains to rendering of the virtual object, and 
in that said first video generation means and said second video generation means generate 
videos of the virtual object using the information that pertains to rendering of the virtual object 
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Claim 4 recites all the limitations of claim 1 and adds the limitation of "information 
generation means" and "generation means generate videos." Ohshima teaches an information 
generation means (e.g., figure 2 and 7; column 3-6; column 11-13). 

Claim 6: 

The apparatus according to claim 1, characterized in that parameters of said first video 
sensing means are known, and said first video generation means generates the video of the 
virtual object viewed from said first viewpoint position in accordance with the known 
parameters. 

Claim 6 recites all the limitations of claim 1 and adds the limitation of "the known 
parameters." Ohshima teaches the known parameters such as the focal length of the camera, the 
width of the imaging surface of the camera (Ohshima column 20). 

Claim 7: 

The apparatus according to claim 1, characterized in that some of parameters of said 
first video sensing means are variable, said apparatus further comprises measurement means for 
measuring changes of the parameters, and said first video generation means generates the video 
of the virtual object viewed from said first viewpoint position in accordance with the parameters 
measured by said measurement means. 

Claim 7 recites all the limitations of claim 1 and adds the limitation of "variable 
parameters" and "measurement means." Ohshima teaches measuring the mallet positions, 
generating and correcting view position and posture of the players (Ohshima column 15-18). 
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Claim 8: 

The apparatus according to claim 7, characterized in that the parameters of said first 
video sensing means measured by said measurement means include at least one of a viewpoint 
position/posture, and zoom ratio. 

Claim 8 recites all the limitations of claim 7 and adds the limitation of "position/posture 
information " Ohshima teaches measuring the mallet positions, generating and correcting view 
position and posture of the players (Ohshima column 15-18). 

Claim 9: 

The apparatus according to claim ], characterized in that when a plurality of first video 
sensing means equivalent to said first video sensing means are present, said apparatus further 
comprises selection means for receiving a plurality of videos of the real space from said first 
viewpoint position from the plurality of first video sensing means, and outputting a video of the 
real space viewed from said first viewpoint position from one selected first video sensing means 
to said first video composition means, and said first video composition means generates a video 
of the virtual object viewed from said first viewpoint position using parameters of the first video 
sensing means selected by said selection means. 

Claim 9 recites all the limitations of claim 1 and adds the limitation of "selection means 
for receiving a plurality of first video sensing means " 

Ohshima teaches selecting of the image regions (Ohshima column 15-18) and cameras 
for sensing a scene in front of them where image signals that represent an environment scene of a 
real space (Ohshima figure 2 and 7; column 3-6; column 11-13). 
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4. Claim 10: 

The claim 10 is a rephrasing of the claim 1 in a method form. The claim is rejected for 
the same reason as set forth in claim 1 . 

Claim 12: 

The apparatus according to claim 10, characterized in that said player 's viewpoint 
augmented reality presentation step further comprises: the second video generation step of 
generating a video of the virtual object viewed from said player 's viewpoint position; and the 
display step of displaying to the player the video of the virtual object viewed from said player 's 
viewpoint position on a display surface through which the player can visually see the real space. 

Claim 12 recites all the limitations of claim 10 and adds the limitation of "the second 
video generation step" and "the display step." 

Ohshima teaches the second video generation step and the display step (Ohshima figure 2 
and 7; column 3-6; column 11-13). 

Claim 13: 

The method according to claim 10, characterized by further comprising information 
generation step of generating information that pertains to rendering of the virtual object, and in 
that said first video generation step and said second video generation step generate videos of the 
virtual object using the information that pertains to rendering of the virtual object 
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Claim 13 recites all the limitations of claim 10 and adds the limitation of "information 
generation step" and "first and second generation steps to generate videos." 

Ohshima teaches the information generation step and the first and second generation 
steps to generate videos (Ohshima figure 2 and 7; column 3-6; column 11-13). 

Claim 14: 

The apparatus according to claim 13, characterized in that said information generation 
step includes the step of generating information of an outer appearance of the virtual object and 
information of a position/posture of the virtual object as the information that pertains to 
rendering of the virtual object 

Claim 14 recites all the limitations of claim 13 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." 

Ohshima teaches generating and correcting view position and posture of the players 
(Ohshima column 18). 
Claim 15: 

The method according to claim 10, characterized in that parameters of said first video 
sensing means are known, and said first video generation step includes the step of generating the 
video of the virtual object viewed from said first viewpoint position in accordance with the 
known parameters. 

Claim 15 recites all the limitations of claim 10 and adds the limitation of "the known 
parameters." Ohshima teaches the known parameters such as the focal length of the camera, the 
width of the imaging surface of the camera (Ohshima column 20). 
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Claim 16: 

The method according to claim 10, characterized in that some of parameters of means for 
sensing a video viewed from said first viewpoint position are variable, said method further 
comprises the measurement step of measuring changes of the parameters, and said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
first viewpoint position in accordance with the parameters measured in the measurement step. 

Claim 16 recites all the limitations of claim 10 and adds the limitation of "variable 
parameters" and "measurement step." 

Ohshima teaches measuring the mallet position, generating and correcting view position 
and posture of the players (Ohshima column 15-18). 

Claim 17: 

The method according to claim 16, characterized in that the parameters of the means for 
sensing a video viewed from said first viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, and zoon ratio. 

Claim 17 recites all the limitations of claim 16 and adds the limitation of 
"position/posture information." Ohshima teaches correcting view position and posture of the 
players (Ohshima column 15-18). 

Claim 18: 

The method according to claim 10, characterized in that when a plurality of means for 
sensing a video viewed from said first viewpoint position are present, said method further 
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comprises the selection step of receiving a plurality of videos of the real space viewed from a 
first viewpoint position from the plurality of means for sensing a video viewed from said first 
viewpoint position, and outputting the video of the real space viewed from a first viewpoint 
position input from one selected means for sensing a video of said first viewpoint position to 
means for compositing a first viewpoint video, and said first video composition step includes the 
step of generating a video of the virtual object viewed from said first viewpoint position using 
parameters of the means for sensing a video viewed from a first viewpoint position selected in 
the selection step. 

Claim 18 recites all the limitations of claim 10 and adds the limitation of "selection step 
for receiving a plurality of videos " 

Ohshima teaches selecting of the image regions (Ohshima column 15-18) and cameras 
for sensing a scene in front of them where image signals that represent an environment scene of a 
real space (Ohshima figure 2 and 7; column 3-6; column 11-13). 



5. Claim 19: 

(1) The claim 19 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of "a storage medium storing a program code. 

(2) It is unclear whether Ohshima has implicitly taught a storage medium storing a 
program code. 

(3) However, Sato discloses the claimed limitation of a storage medium that stores an 
image processing program, which is implemented on a computer and continuously presents 
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three-dimensional images to an observer/player, storing a program code of an augmented reality, 
a depth estimation program code, a depth image generation program code, a position/posture 
information estimation program code, a warping program code and a program code of presenting 
to the observer three-dimensional images. The Sato reference implicitly teaches a program code 
including the augmented reality presentation step of generating and presenting a virtual image in 
a real space in which an image of merged real objects and virtual objects are displayed in LCDs 
103 in an optical see-through head mounted device (column 1, lines 13-23). The Sato reference 
also implicitly teaches a position/posture estimation module 201 that outputs three-dimensional 
motions from a viewpoint position of the camera to right and left viewpoint positions of the 
player (column 12, lines 54-65). The Sato reference further teaches a depth image generation 
module 300 that uses position/posture information input from the position/posture estimation 
module 201 as that for CG rendering and which generates an augmented reality image using the 
three-dimensional CG database in accordance with the distance to an object in the real world 
expressed by the depth image and presents it on the LCDs 103 (column 12, lines 44-50) and a 
depth warping module 203 to inversely project a depth image ID acquired at a viewpoint having 
position/posture information into a space, and to re-project it onto the imaging plane of the 
virtual camera with the focal length of the virtual camera assumed at the viewpoint having an 
estimated position/posture value by the viewpoint position/posture module 201 (column 13, lines 
32-48). 

(4) Ohshima at least suggests program codes illustrated in a variety of flow charts (See 
Ohshima figure 7, 8, 9, 10, 14, 16, 17, 23). 
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(5) One having the ordinary skill in the art would have been motivated to do this because 
it would have provided the installation and implementation of the image processing on other 
systems. 

Claim 24: 

The medium according to claim 19 \ characterized in that parameters of means for 
sensing said first viewpoint video are known, and the program code of said first video generation 
step includes the step of generating the video of the virtual object viewed from said first 
viewpoint position in accordance with the known parameters. 

Claim 24 recites all the limitations of claim 19 and adds the limitation of "the known 
parameters." 

Ohshima teaches the known parameters such as the focal length of the camera, the width 
of the imaging surface of the camera (Ohshima column 20). 

Claim 25: 

The medium according to claim 19, characterized in that some of parameters of means 
for sensing a video viewed from said first viewpoint position are variable, the program code of 
said medium further comprises the measurement step of measuring changes of the parameters, 
and the program code of said first video generation step includes the step of generating the video 
of the virtual object viewed from said first viewpoint position in accordance with the parameters 
measured in the measurement step. 
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Claim 25 recites all the limitations of claim 19 and adds the limitation of "variable 
parameters" and "measurement step." 

Ohshima teaches measuring the mallet positions, generating and correcting view position 
and posture of the players (Ohshima column 15-18). 

Claim 26: 

The medium according to claim 25, characterized in that the parameters of the means of 
sensing a video viewed from said first viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, andzoon ratio. 

Claim 26 recites all the limitations of claim 25 and adds the limitation of 
"position/posture information." 

Ohshima teaches measuring the mallet positions, generating and correcting view position 
and posture of the players (Ohshima column 15-18). 

Claim 27: 

The medium according to claim 19, characterized in that when a plurality of means for 
sensing a video viewed from said first viewpoint position are present, said medium further 
comprises a program code of the selection step of receiving a plurality of videos of the real 
space viewed from a first viewpoint position from the plurality of means for sensing a video 
viewed from said first viewpoint position, and outputting the video of the real space viewed from 
a first viewpoint position input from one selected means for sensing a video of said first 
viewpoint position to means for compositing a first viewpoint video, and the program code of 
said first video composition step includes the step of generating a video of the virtual object 
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viewed from said first viewpoint position using parameters of the means for sensing a video 
viewed from a first viewpoint position selected in the selection step. 

Claim 27 recites all the limitations of claim 19 and adds the limitation of "selection step 
for receiving a plurality of videos." 

Ohshima teaches selecting of the image regions (Ohshima column 15-18) and cameras 
for sensing a scene in front of them where image signals that represent an environment scene of a 
real space (Ohshima figure 2 and 7; column 3-6; column 11-13). 

6. Claims 28-29: 

Referring to claims 28-29, Ohshima discloses a mixed reality presentation system that 
generates and presents a virtual image in a real space (see figures 2 and 7 of Ohshima). However, 
the reference is silent on a printing means in connection to the mixed reality presentation 
apparatus. It is common that a computer system has a printing means attached to them. 
Therefore, it would have been obvious to one having ordinary skill in the art to have 
incorporated a printing means in the mixed reality presentation apparatus of Ohshima because 
such construction is rather conventional. A person of ordinary skill in the art would be motivated 
to have incorporated a printing means to the Ohshima' s mixed reality presentation apparatus to 
further provide a paper copy of still images of the real images that have been displayed on the 
display devices of Ohshima. 



7. Claim 31: 
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The claim 3 1 encompasses the same scope of invention as that of claim 1 . The claim is 
subject to the same reasons set forth in claim 1. 

8. Claim 32: 

The claim 32 encompasses the same scope of invention as that of claim 1. The claim is 
subject to the same reasons set forth in claim 1 . 

Remarks 

9. Applicant's arguments, filed 1 1/20/2003, paper number 1 1, have been fully considered 
but they are not deemed to be persuasive. 

10. Applicant argues in essence with respect to the amended Claim 1 and similar claims that: 
a. "Claim 1 has been amended as discussed during the interview, to further clarify 
that the term 'independently 5 means -separate-. Also, the term 'any' has been inserted to 
clarify that, if more than one player is present, the first viewpoint position cannot be the 
viewpoint of another player. ..For the foregoing reasons, Applicants submit that this 
application is in condition for allowance" 

In response, the Examiner asserts that Ohshima/Latypov/Sato meets the claim limitations 
set forth in the amended Claim 1. Although Ohshima lacks a full disclosure of display means for 
displaying the mixed reality videos on a predetermined display apparatus that is separate from 
any player's display apparatus . Latypov teaches display means for displaying the augmented 
reality videos including displaying the image in virtual space on a predetermined display 
apparatus and the superimposed image from the real space and virtual space on a separate 
display apparatus (e.g., Latypov column 1 1). However, Ohshima at least suggests display means 



• 
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for displaying the augmented reality videos on a predetermined display apparatus (displaying the 
image on a table in a real environment) that is separate from any player's display apparatus (See 
Ohshima figure 7; column 11, lines 50-65; column 12-13) and Ohshima teaches display means 
for displaying the augmented reality videos on a display apparatus (displaying the image on a 
table in a real environment or displaying the mixed reality image on other player's display 
apparatus or displaying the mixed reality image on the left/right-hand side HMD) that is separate 
from ONE of the player's display apparatus (See Ohshima figure 7; column 1 1, lines 50-65; 
column 12-13). One having the ordinary skill in the art would have been motivated to do this 
because it would have provided a separate display device for viewing by the players. Therefore, 
Ohshima/Latypov/Sato fulfills the amended Claim 1 as currently drafted. 



1 1 . Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

12. A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 

CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 



Conclusion 
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however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

13 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 308-6606 for regular 
communications and (703) 308-6606 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 395-3900. 



jew 

December 30, 2003 




PRIMARY EXAMiNER 



